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Abstract  -  VR  (Virtual  Reality)  system  have  been  widely  used 
for  various  purposes.  However,  during  people’s  immersing  in  a 
virtual  environment  it  is  commonly  reported  that  simulation 
sickness  can  occur,  and  it  prevents  us  from  utilizing  a  VR 
environment  for  the  wider  purposes. 

We  constructed  controlled  a  VR  environment  for  analyzing 
the  change  of  bio-signals  during  VR  immersion,  where  subjects 
were  requested  to  find  trash  cans  in  the  virtual  environment 
within  five  minutes.  Each  subject’s  various  bio-signals,  which 
were  EEGs  from  5  different  locations,  vertical  EOG,  Lead  I 
ECG,  fingertip  skin  temperature,  photoplethysmogram,  and 
skin  conductance  level,  were  measured  during  experiments. 
We  analyzed  and  compared  the  signals,  and  we  found  out  that 
the  characteristics  of  28  signals  during  nausea  were  statistically 
different  from  when  the  subjects  were  at  rest,  or  during  the 
first  30  seconds  after  the  immersion  was  started. 

We  parameterized  these  characteristics  and  established  12 
principal  components  using  principal  component  analysis  in 
order  to  reduce  the  redundancy  in  those  parameters,  and 
constructed  an  artificial  neural  network  with  those  principal 
components.  Using  the  network  we  constructed,  we  could 
partially  detect  nausea  in  real  time. 
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I.  Introduction 

VR  (Virtual  Reality)  technology  has  been  used  for  various 
applications,  such  as  flight  simulation  [1],  driving  simulation 
[2],  and  rehabilitation  system  for  patients  who  suffer  from 
acrophobia  [3],  etc. 

However,  during  immerging  in  a  virtual  reality 
environment,  many  people  reported  simulation  sickness, 
such  as  nausea,  vomit,  headache,  etc.  This  simulation 
sickness  prevents  virtual  reality  from  being  used  widely. 
Simulation  sickness  has  been  studied  by  many  researchers. 
For  example,  Regan  et  al.  studied  the  side  effects  of 
immersion  in  virtual  reality  [4].  We  also  note  Regan  et  al.’s 
article  about  reducing  the  side  effects  induced  during 
immersion  in  virtual  reality  using  Hyoscine  Hydrobromide 
[5]. 

Our  research  focuses  on  the  system  that  can  give  feedback 
when  it  finds  a  subject  who  in  experiencing  a  VR 
environment  feels  nausea;  such  a  system  should  also 
minimize  the  inconvenience  by  use  of  electrodes.  In  order  to 


reduce  simulation  sickness,  the  system  measures  and 
analyzes  the  bio-signal  of  the  subject  who  is  immerged  in  a 
virtual  reality  environment,  finds  out  by  using  artificial 
intelligent  system  whether  the  subject  feels  nausea  or  not, 
and  gives  a  feedback  to  the  subject  when  the  system  detects 
that  the  subject  feels  nausea.  To  minimize  the  inconvenience 
caused  by  the  electrodes  special  kinds  of  electrodes  should 
be  developed,  such  as  an  EEG  and  EOG  electrode  equipped 
in  HMD  (Head  Mounted  Display),  and  a  skin  temperature 
and  phoplethysmogram  electrode  equipped  in  glove.  The 
primary  aim  of  this  study  is  to  give  a  preliminary  guide  for 
the  research. 

II.  Methods 

1.  Experiment  for  analyzing  the  bio-signal  of  a  subject  who 
is  immerging  in  a  VR  environment 

1.1  Subjects 

45  subjects  who  were  18-26  year-old  college  students 
participated  in  the  experiment.  (25  Male,  20  Female,  Mean 
21.9  yr.)  All  of  these  subjects  had  intact  vestibular  function 
and  were  not  receiving  medication  at  the  time. 

1.2  Equipment 

The  VR  system  used  for  this  experiment  was  a  3D  visual 
and  auditory  environment  generator  (3  channels,  resolution: 
3840x1024,  constant  30  frames/sec),  constructed  by  the 
KIST  (Korean  Institute  of  Science  and  Technology). 

Various  kinds  of  bio-signal  electrodes  were  attached  to 
the  subjects.  These  electrodes  measured  EEGs  from  5 
different  locations  Fz,  Cz,  Pz,  Ol,  02),  vertical  EOG,  Lead  I 
ECG,  fingertip  skin  temperature,  photoplethysmogram,  and 
skin  conductance  level.  Using  MP-100  by  BIOP  AC  Co.  we 
sampled  the  bio-signals  of  all  10  channels  in  200Hz 
sampling  rate  and  stored  them  in  a  computer. 
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Table i 

PERIODS  USED  FOR  DATA  ANALYSIS 

Periods 

Details 

Before  VR 
(Period  I) 

From  when  the  experimental  setting  is  completed 
for  each  subjects  to  when  VR  simulation  starts 
(1  minute) 

Right  after  VR 
(Period  II) 

For  30  seconds  after  VR  simulation  starts 

During  VR 
(Period  III) 

From  30  seconds  after  VR  simulation  starts  to  the 
end  of  VR  simulation 

1.3  Procedure  of  experiment 

Before  each  subject  experienced  the  VR  environment 
his/her  bio-signals  were  measured  for  1  minute,  and  he/she 
experience  VR  environment  for  5  minutes  each. 

The  subjects  were  requested  to  find  trashcans  in  the 
virtual  background  of  KIST  buildings  and  to  report  nausea 
to  us  orally  whenever  the  felt  sickness. 

2.  Data  Analysis 

29  out  of  the  total  45  subjects  reported  nausea  at  least 
once.  We  analyzed  the  signals  of  these  29  subjects.  The 
analysis  was  performed  within  3  different  periods  (Table  I): 
before  experiencing  VR  simulation  for  a  minute;  right  after 
VR  started  for  30  seconds;  and  from  30  seconds  after  VR 
started  to  the  end  of  VR. 

For  Period  III  nausea  and  normal  period  were  classified. 
The  nausea  period  is  within  3  seconds  at  the  time  when  a 
subject  reports  nausea,  and  we  divided  the  whole  period  III 
into  3-second  periods.  [Normal  periods  are  those  3-second 
periods  that  do  not  contain  any  nausea  points.] 

3.  Feature  Extraction 

We  extracted  features  in  each  period  from  each  subject’s 
bio-signal  as  in  the  Table  h. 

Firstly,  we  calculated  the  relative  power  of  each 
frequency  band.  These  ranged  from  theta  (5-8Hz),  alpha 
(9-13Hz),  beta  (14-30Hz),  and  gamma  (31-50Hz)  from 
EEGs  in  5  different  locations.  The  relative  power  of  each 
frequency  band  is  related  with  mental  activities  [6], 

Secondly,  we  calculated  the  standard  deviation  of  EOG. 
Eye  blink  frequency  is  related  with  highly  cognitive  mental 
process  [7].  The  standard  deviation  of  vertical  EOG  contains 
eye  blink  information. 

Thirdly,  the  mean  of  R-R  intervals  was  calculated  from 
raw  ECG  using  BIOPAC  MP-100  software.  Heart  rate  is 
related  with  mental  response.  The  ANS  (autonomic  nervous 
system)  causes  the  heart  to  beat  faster  or  slower  according  to 
mental  response. 


Table  n 


ANALYZED  FEATURES 


Electrodes 

Features 

EEGs  from  5  different  locations 
(Fz,  Cz,  Pz,  01,  02) 

Relative  power  of  each 
frequency  bands 

-  Theta  (5-8Hz) 

-Alpha  (9-13Hz) 

-  beta(14-30Hz) 

-  gamma  (31-50Hz) 

EOG 

Standard  deviation 

ECG 

R-R  interval  mean 

Finger  tip  skin  temperature  (SKT) 

-  mean 

-  standard  deviation 

Photoplethysmogram  (PPG) 

-  mean 

-  standard  deviation 

Skin  conductance  (SCL) 

-  mean 

-  standard  deviation 

Fourthly,  we  calculated  the  mean  and  standard  deviation 
of  fingertip  temperature.  Skin  temperature  is  linked  to 
sympathetic  arousal  that  in  turn  affects  vasoconstriction.  The 
latter  affects  the  perfusion  of  blood,  particularly  in 
extremities  such  as  the  fingers  and  toes.  Typically, 
sympathetic  arousal  leads  to  increased  vasoconstriction, 
which  causes  a  reduction  in  blood  volume  and  hence  to  a 
cooling  effect  on  the  skin.  [8] 

Fifthly,  we  calculated  the  mean  and  standard  deviation  of 
photoplethysmogram.  Photoplethysmogram  measures  blood 
volume  in  fingertip. 

Sixthly,  we  calculated  the  mean  and  standard  deviation  of 
skin  conductivity.  The  latter  has  been  recognized  as 
distinctively  sensitive  to  transitory  emotional  states  and 
mental  events,  while  often  remaining  more  or  less 
independent  of  other  measures  such  as  muscle  tension  and 
skin  temperature. [8] 

4.  Feature  Analysis 

In  order  to  detect  nausea  from  each  subjects  in  real-time 
the  parameters  used  for  detection  should  be  independent  of 
the  characteristics  of  the  subjects;  thus  the  features  should  be 
duly  modified  for  this  purpose.  Each  subject  has  a  different 
base  line  for  each  feature,  which  makes  it  difficult  for  an 
artificial  neural  network  (ANN)  to  be  trained.  Therefore  we 
defined  the  base  line  value  of  each  features  as  feature  values 
in  Period  I  or  Period  II  according  to  the  characteristics  of  the 
features.  The  ratios  of  feature  values  in  the  3-second  time 
window  to  the  base  line  are  selected  were  parameters 
(Table  m). 

We  analyzed  these  parameters  with  T-test  and  obtained  a 
result  that  the  parameter  values  when  a  subject  feels  nausea 
are  statistically  different  from  the  values  in  normal  state. 
(p<0.05) 


Table  m 


RELATED  PERIODS  FOR  EACH  FEATURE 

Features 

Related  period  according  to  their 
characteristics 

All  EEG  frequency  bands, 

PPG  Standard  deviation 

Period  I 

PPG  Mean 

EOG  Standard  deviation 

GSR  Mean 

GSR  Standard  deviation 

Heart  Beat  Rate 

Period  11 

5.  Construction  of  an  artificial  neural  network 

Training  vectors  were  constructed  from  parameters  in 
both  nausea  and  normal  period.  Nausea  is  a  scarce  event 
compared  with  a  normal  event  so  we  repeated  nausea 
vectors  equivalent  to  the  normal  vectors.  There  exists  some 
inherent  redundancy  in  28  parameters  because  each 
frequency  band  in  the  EEGs  has  similar  information  for 
nausea.  In  order  to  eliminate  this  redundancy  principal 
component  analysis  was  performed.  We  obtained  12 
principal  vectors  which  could  be  used  as  an  input  vector  of 
the  artificial  neural  network. 

We  trained  2-layer  feedforward  neural  network  whose 
hidden  layer  has  10  nodes  with  standard  error  backprogation 
algorithm.  The  minimum  mean  square  error  we  got  was 
0.092. 

6.  Construction  of  real-time  nausea  detection  system 

At  first  we  collected  parameters  of  Period  I  and  Period  II  - 
that  is  30  seconds  after  the  start  of  VR.  After  they  were 
collected  a  parameter  for  the  previous  3  seconds  could  be 
calculated  per  second.  Ratios  of  each  parameter  calculated  to 
Period  I  or  Period  II  can  be  calculated;  these  ratios  would  be 
transformed  with  a  transform  matrix  attained  from  principal 
component  analysis  of  training  vectors.  Those  transformed 
ratios  would  be  inputted  to  the  ANN  which  we  constmcted, 
and  we  would  find  out  whether  the  signal  is  related  with 
nausea  or  not. 

III.  Result  and  discussion 

We  tested  the  validity  of  the  real-time  nausea  detection 
system  with  the  entire  time  series  of  each  subject.  Fig.  1 
shows  the  comparison  between  the  system  output  and  actual 
nausea  points.  As  seen  in  the  picture,  the  ANN  output  and 
actual  nausea  points  are  not  identical.  The  reason  is  because 
nausea  was  reported  verbally  so  that  the  actual  points  could 
be  inaccurate,  and  the  nausea  level  is  not  standardized  for 
each  subject.  So  some  subjects  may  report  nausea  when  light 
symptom  occurs  while  others  may  report  nausea  only  when 
heavy  symptoms  occurs.  Another  factor  is  that  the  ANN 
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Fig.  1  Comparison  of  ANN  out  put  with  Actual  Nausea  points 

parameters  we  used  as  inputs  were  not  very  good  ones;  as  a 
consequence  nausea  might  not  be  detected  very  well.  The 
minimum  mean  square  error  of  the  ANN  training  set  was 
0.09.  This  could  be  improved  if  we  chose  better  parameters. 

In  the  future  we  aim  to  construct  a  better  experiment 
system  and  to  study  other  parameters  for  the  better 
performance  of  the  nausea  detection  system. 
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